Survey on Comparable Corpora until June 2012

Here we present a survey of important work done on Comparable Corpora between the period 1995 to 2012. Unlike parallel corpora, which are clearly defined as translated texts, there is a wide variation of non-parallelism in comparable text. Non-parallelism is manifested in terms of differences in author, domain, topics, time period, language. The most common text corpora have non-parallelism in all these dimensions. The higher the degree of non-parallelism, the more challenging is the extraction of bilingual information. Such a corpus is nevertheless a desirable source of bilingual information, especially for new words. In this report we have first classified the research on comparable corpora into various categories. This is followed by detailed literature survey on comparable corpora and comparability metrics. After that we discuss the work related to the enhancement of comparability metrics in corpus. We conclude with the brief summary of this survey on comparable corpora. 1 Classification of Work on Comparable Corpora We can broadly classify the research on comparable corpora into the following sections. • Correlation based extraction • Vector representation • Classifiers based extraction • Linguistic knowledge based extraction Each of these classes are described in the following sections along with the gist of pioneering work in these domain. 2 Correlation based Extraction Most of the work on comparable corpora is based on correlation between word co-occurrence. They consider the context of a word as a feature to map the source word to the target word. Moreover most of the work based on this idea is focused towards extraction of bilingual lexicons only rather than parallel sentences.